Speech-to-lip movement synthesis based on the EM algorithm using audio-visual HMMs

This paper proposes a method to re-estimate output visual parameters for speech-to-lip movement synthesis using audio-visual Hidden Markov Models(HMMs) under the Expectation-Maximization(EM) algorithm. In the conventional methods for speech-to-lip movement synthesis, there is a synthesis method estimating a visual parameter sequence through the Viterbi alignment of an input acoustic speech signal using audio HMMs. The HMM-Viterbi method produces the output visual parameters per HMM state speci ed by the decoded HMM states. However, the HMM-Viterbi method involves a substantial problem, which is caused by the deterministic decoding process to assign a single HMM state for an input audio frame. The deterministic process may output incorrect visual parameters due to incorrect HMM state alignment. The proposed method avoids the deterministic decoding process by the non-deterministic visual parameter estimation by the EM algorithm. The proposed method repeatedly estimates the visual parameter sequence while maximizing the likelihood of the audio-visual observation sequence using audio-visual HMMs. The objective evaluation shows that the proposed method is more e ective than the HMM-Viterbi method especially for the bilabial consonants.